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ABSTRACT 

In this study, a programmatic decision-making 
framework called the consensus marker-outcome variable system (CM0”-)- 
was developed as a result of a coi"pr&l;*;asive "meta-review" and 
synthesis of research on variables considered by experts to be 
important to learning for all students, including those with special 
needs or those at risk of failing. The CMOVS systematically analyzes 
program design and implementation features, assesses them in relation 
to site-specific needs, and provides a basis for calculating 
informational indexes. The information derived from the CMOVS can 
then be used to assist stakeholders in programmatic decision making. 
The first section of the paper provides an overview of the CMOVS 
framework, which groups 228 identified variables into 6 categories: 

(1) state and district variables; (2) out of school contextual 
variables; (3) school-level variables; (4) student variables; (5) 
program design variables; and (6) implementation, classroom 
instruction, and climate variables. The second section focuses on the 
research base for the design of the decision-making framework 
including the literature review and a Delphi survey of experts and 
practitioners. The final section provides detailed sample 
illustrations of the use of the CMOVS as a conceptual guide for 
making programming decisions. This section also shows how the CMOVS 
can be used to develop a Program Effectiveness Index and a 
Desirability of Implementation Index. Appended are the master list of 
variables; definitions; consensus from the field; and a CMOVS 
computer analysis of user ratings of selected, innovative educational 
practices. (Contains 13 references.) (Autor/DB) 
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Abstract 



A programmatic decision-making framework, the consensus 
marker-outcome variable system (CMOVS), was developed as a result 
of a comprehensive "meta-review" and synthesis of research on 
variables considered by experts to be important to learning. The 
CMOVS systematically analyzes program design and implementation 
features, assesses them in relation to site-specific need, and provides 
a basis for calculating informational indexes. The information 
derived from the CMOVS can then be used to assist stakeholders in 
programmatic decision making. 
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A DECISION-MAKING FRAMEWORK 
FOR DESCRIPTION 

OF INNOVATIVE EDUCATION PROGRAMS 



This paper discusses a decision-making framework concerning the design 
and implementation features of innovative educational practices/ programs. 
The framework, derived from the findings of a study designed to obtain a 
consensus on variables that are important to learning (Wang, Walberg, 
Reynolds, & Rosenfield, 1989), aims to provide a conceptual basis for 
systematic analysis, description and identification of features and 
implementation requirements of innovative educational programs for improving 
instruction and learning in regular classroom settings. 

The first section provides an overview of the decision-making framework; 
the second section focuses on the research base for the design of the 
decision-making framework; and the final section provides sample illustrations 
of the use of the CMOVS as a conceptual guide for making programming 
decisions. 



The Decision-Making Framework 

The decision-making framework discussed in this paper, the Consensus 
Marker-Outcome Variable System (CMOVS), incorporates variables that are 
considered by professionals as important to learning, based on a recently 
completed research synthesis study (Wang, 1990). The CMOVS was developed 
with the goal of providing a common language that can be used by researchers 
to align concepts and methodologies across studies concerning variables that 
are important to learning. It also provides a synthesis of research findings 
that can be used by practitioners and policy makers to improve communication 
about programs, their features, and their implementation requirements for 
planning, documentation, and decision-making. 

The impetus for the development of the decision-making framework arose 
out of two specific concerns about the current state of practice. The first is 



concern about the quality of education programs in terms of how they respond 
to increasing diversity among students. The second is concern about the need 
to develop a systematic information base on how to use what we know works 
to improve schools’ capabilities to achieve the educational vision of providing 
equity in learning outcomes for all students, including those with special needs 
and/or those considered to be at risk of failing or dropping out of school. 

Concern for Quality Education 

The CMOVS was conceived within the context of rising public concern 
over the general quality of education, and in particular, the effectiveness of 
current educational approaches for students with special needs or otherwise 
considered educationally at risk -- those who require greater-than-usual 
educational and related service support. This concern for educational 
effectiveness has been expressed in a multitude of reports by a variety of 
commissions and study groups (e.g., Carnegie Forum on Education and the 
Economy, 1986; Committee for Economic Development, 1985; Council of Chief of 
State School Officers, 1987; Hawkins, 1986; National Coalition of Advocates for 
Students, 1985; National Governors’ Association, 1986). There is a clear 
mandate to improve the school’s ability to effectively and efficiently serve all 
students, inciting those who require special education or other remedial or 
compensatory programs, as well as those otherwise considered to be at risk of 
either failing or dropping out of school. 

The Need to Build a Systematic Information Base 

While there have been major efforts toward reform, current practices fall 
woefully short of this mandate. There is no lack of information on what to 
do to improve current practice (cf. Wittrock, 1986; Wang, Reynolds & Walberg* 
1987-1989). However, there is a significant lack of systematic information on 
what we know works and how to use what we know to improve instruction and 
learning in schools. 

Local schools face two demanding tasks: first, obtaining information on 

the design, implementation requirements* and efficacy of innovative 
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approaches/practices; and second, evaluating the feasibility and the site- 
specific compatibility of the approaches with the objectives of a particular 
school district and/or school. Findings from a 1983 survey (Research for 
Better Schools) showed that local educators in New Jersey and Pennsylvania, 
for example, look to external training and technical assistance programs for 
staff development and support in three major areas: (a) curriculum and 

instruction, especially in terms of the implementation of innovative programs 
and practices, development and improvement of curricula, and in-service staff 
development for school personnel; (b) administration, including organizational 
planning, staffing and scheduling, facilities maintenance, and management skills, 
such as instructional leadership and communication; and (c) knowledge about 
the "outside world," including state and federal regulations and community 
relations. 

Presently, there are few tools available to assist local schools/school 
districts in selecting approaches/practices for meeting their specific needs. 
This lack has resulted in a limited ability of school personnel to make 
informed decisions in selecting practices/programs, that is, how such decisions 
are aligned with local educational goals, resources, and needs. The CMOVS is 
intended to provide a systematic framework for guiding analysis of program 
design and implementation features in ways that can be helpful to school 
personnel and policy makers in developing, identifying, and selecting innovative 
practices/programs. 

The Development of the Consensus Marker-Outcome Variable System 

The variables included in the CMOVS are considered to be both important 
to learning and, perhaps more significant, "alterable" (educators have some 
chance of changing them in ways that enhance learning), thereby improving 
chances for students’ learning success. In other words, variables included in 
the CMOVS are concerned with learning conditions that can bring about 
educational outcomes for students. Thus, in a real sense the consensus 
represented in the CMOVS is reflective of the recent shift from the study of 
"static" variables that are not easily alterable by schools (e.g., sex, age, SES, 
history of education) to the study of instruction and learning as they take 
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place under specific environmental conditions. 

Specifically, development of the CMOVS was based on the use of 
contemporary professional literature and expert opinions’ to answer the 
following questions: What aspects of school and instruction enhance student 

learning? What kinds of social relationships are important to enhance student 
learning in regular classroom settings? What learner characteristics are 
important and alterable in improving learning of students with special needs? 

In order to specify the well-confirmed knowledge about school learning, 
the development of the CMOVS began with a comprehensive meta-review and 
synthesis of research on variables considered to be important to learning. 
Then, various groups of educational professionals were asked to make 
judgments about the importance of the identified variables or principals in 
their work. Thus, the first step involved a detailed reading of the professional 
literature to make a "first approximation" list of important variables based on 
a conceptual framework of variables (Wang, 1986) that are important to 
learning in school contexts. Figure 1 shows a schematic representation 'of the 
conceptual framework. 

Selection of a corpus of studies for analysis and synthesis 

A vast research literature addresses one or more of the potential learning 
influences represented by the conceptual framework, and it clearly would not 
be possible to examine all of the thousands of original studies relevant to a 
synthesis of this scope. Indeed, even the literature of review articles is 
massive. For this reason we focused on authoritative reviews, handbook 
chapters (especially those sponsored by the American Educational Research 
Association and other organizations), selected additional syntheses in 
government documents and other sources. 

A preliminary list of sources was reviewed by the Scientific Advisory 
Panel, and revised after their recommendations. Following this review, the 
sources chosen included chapters from the past decade or more of the Review 
of Research in Education, the Annual Review of Psychology, and the Annual 
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Review of Sociology, as well as the Handbook of Research on Teaching 
(Wittrock, 1986), Designs for Compensatory Education (Williams, Richmond & 
Mason, 1986), more specialized handbooks, and a small number of journal 
articles chosen to assure coverage of all the areas addressed in the 
comprehensive framework. Initially, over 200 articles, chapters, and other 
sources were identified. All of these sources were read, but some were 

excluded from the final corpus because they failed to address K-12 instruction 
in regular school settings, because they addressed exceptionally narrow and 
atypical learning outcomes, or because they were relevant only to rare or 
special-learner populations. 

A total of 179 sources were included in the final corpus of studies (86 
chapters from annual review series, 44 handbook chapters, 20 government and 
commissioned reports, 18 book chapters, and 11 review articles). All of these 
were relevant to a range of cognitive and/or affective learning outcomes for 
K-12 learners in formal educational settings. A list of more than 200 variables 
was assembled based on the literature reading. A detailed analysis of the 
literature is included in a paper on variables that are considered important to 
learning (Wang, Haertel, and Walberg, 1989). 

In the next step, the Delphi survey technique was used to survey expert 
opinions about variables that are considered to be important to learning. A 
full report of the survey finding is in Wang, Reynolds, Walberg and Rosenfield, 
(1989). Briefly, a panel of 12 experts was identified for this phase of 
information gathering. The expert panel included leading researchers as well 
as outstanding practitioners and editors of professional journals. Using a scale 
from 1 (low) to 3 (high), the panel was asked to rate the importance of each 
of the variables in terms of demonstrated importance to student learning. 
They were also asked to add, delete, and suggest changes on the list of 
variables. In accordance with Delphi procedures, results of the "first round" 
of ratings were then sent back to the experts and they responded in a "second 
round," taking into account what other experts had said in the first round. 
Responses in both rounds formed the basis for revising the survey instrument, 
the survey of variables considered important to learning, which was then sent 
to a broad sample of professionals in order to form the data base for the 
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development of the CMOVS. 

Findings from a Survey of Consensus from the Field 

To investigate questions about consensus among various educators on 
alterable variables considered important, eight groups of professionals were 
identified and asked to respond to the survey of variables considered important 
to learning. Through the cooperation of the Council for Exceptional Children 
(CEC) a random sample of 1001 teacher members of CEC was obtained; all are 
special educators. Of that sample, 449 (45%) responded to the survey. Each 
of the special education teachers was asked to recruit as an additional 
respondent the "regular" teacher whose classroom was nearest to his/her own 
classroom : 182 regular teachers responded. 

In addition, a sample of 526 school psychologists was selected randomly 
from the membership list of the National Association of School Psychologists; 
207 responded. Each psychologist was asked to recruit a school principal in a 
building they served . Fifty school principals responded. All state directors of 

special education and of Chapter I programs were asked to complete the 

survey, which they did at relatively high rates: 64% (N - 36) and 58% (N - 

40), respectively. A group of special education researchers was identified by 
assembling names of recipients of federal research grants in the field of 
special education, specifically relating to services for mildly handicapped 
students in regular education settings; 55 of 197 responded. A final category 
of educational researchers/authors was created by assembling names of first 
authors of 134 major chapters in the professional literature used in the "meta- 
review" that initiated the development of the CMOVS; 61 (46%) responded. 

To determine the extent of agreement among various groups of 
respondents to the survey on variables considered important to learning, 
Pearson correlations among the mean ratings of items as determined by the 
eight educator groups were examined. The results, summarized in Table 1, 
suggest a very high degree of consensus among such educator groups on 
variables that are important in attempting to enhance the learning of children 
in school. It may be noted, for example, that the correlation of mean ratings 



Table 1 



Pearson Correlations of Mean Ratings of 228 Variables by Eight Respondent Groups 



Respondent Groups 


ERA 


SER 


SPs 


SPr 


SDSE 


SDCI 


RET SET 


Educ. Researchers/Authors (ERA) 


1.00 














Special Educ. Researchers (SER) 


.91 


1.00 












School Psychologists (SPs) 


.88 


.90 


1.00 


• 








School Principals (SPr) 


.84 


.86 


.93 


1.00 








St. Dtrs. of Special Educ. (~DSE) 


.77 


.87 


.89 


.87 


1.00 






St. Dtrs. of Ch. I Programs (SDCI) 


.81 


.84 


.92 


.92 


.88 


1.00 




Regular Educ. Teachers (RET) 


.80 


.82 


.92 


.94 


.82 


.89 


1.00 


Special Educ. Teachers (SET) 


.78 


.85 


.95 


.92 


.88 


.89 


.95 1.00 
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by regular and special education teachers was .97. Though this was the 
highest correlation observed, all correlations tended to be high, with the 
median among 28 correlations being .88. The lowest correlation (.80) was 
between State Directors of Special Education and Education Researchers/ 
Authors. Judged by correlational analysis, there is remarkable similarity in the 
views of special and regular education teachers about principles to be 
considered in their teaching. 

To further analyze the consensus among the eight educator groups, the 
mean ratings of the items that received the highest and lowest scores in each 
of the groups were examined. Even though the correlations across groups 
were high, when considering all 228 items in the survey, it was thought that 
the top-rated items might be somewhat different among groups. Table 2 
displays items that received a mean rating of 2.7 or above, and Table 3 
displays ratings below 1.9. Some contrasting patterns in the mean rating by 
groups are noted. It seems that there is more consensus among the groups on 
the lowest ranked items (see Table 3). However, some consistent patterns of 
differences were suggested in the data (see Table 2). For example, the 
patterns of ratings among principals, regular and special education teachers, 
and Chapter 1 directors were more similar when compared to those of the 
researchers. On the other hand, the ratings of special and regular education 
researchers and state directors of special education were even more alike. 

Discussion of Study Findings 

Although conclusions of consensus yield from a synthesis of the research 
base and the Survey of Consensus from the field, certain caveats should be 
noted. For example, it cannot be determined from the analyses of the 
literature what actual effect sizes will result; the analyses merely estimate 
their relative sizes. In addition, the analyses yield neither actual nor relative 
estimates of combinations of practices. It would seem reasonable to suppose 
that implementation of more practices with the highest estimates would yield 
the largest effects, but this supposition is a matter for subsequent empirical 
research. 
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Table 2 

Patterns of Differences in Mean 
Ratings at or Above 2.7 Among Stakeholder Groups 



Variables with Mean Ratings 
at or Above 2.7 by Total & Group 






Mean Rating by Stakeholder Groups 






TOTAL 

(1123) 


RER 

(61) 


SER 

(55) 


PSY 

(207) 


PRN 

(91) 


SD 

(37) 


ClD 

(41) 


RET 

(182) 


SET 

(449) 


Out School Contextual Variables 




















Home Environment & Parental Support 




















parental involvement in assuring completion of homework 


273 








X 




X 


X 


X 


parental involvement in assuring attendance 


285 






X 


X 


X 


X 


X 


X 


parental application of discipline 


277 






X 


X 




X 


X 


X 


parental interest in student’s work 


283 


X 


X 


X 


X 


X 


X 


X 


X 


parental expectation for academic success 


280 


X 


X 


X 


X 




X 


X 


X 


School Level Variables 




















Teacher! Administrator Decision Making 




















teacher involvement in instructional decision making 


275 




X 


X 


X 




X 


X 


X 


teacher involvement in increasing academic performance 


279 




X 




X 


X 


X 


X 


X 


School Culture (Ethos) 




















safe, orderly school climate 


275 








X 




X 


X 


X 


Student Variables 




















Social and Behavioral 




















positive behavior 


280 






X 


X 






X 


X 


cooperativeness with teachers 


271 








X 






X 


X 


Motivational and Affective 




















attitude toward school 


285 






X 


X 




X 


X 


X 


altitude toward teachers 


277 






X 


X 




X 


X 


X 


motivation for continual learning 


282 




X 




X 


X 


X 


X 


X 


perseverance on learning tasks 


276 






X 


X 




X 


X 


X 


self confidence 


279 




X 


X 


X 


X 


X 


X 


X 


Cognitive 




















level of reading comprehension ability 


279 


X 


X 


X 


X 


X 


X 


X 


X 


level of listening skills 


276 


X 




X 


X 




X 


X 


X 



Note: RER - regular ed. researcher SD - state director 

SER - special ed. researcher C1D - chapter 1 director 
PSY - psychologist RET - regular ed. teacher 

PRN • principal SET - special ed teacher 
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Table 2 



(continued) 



Variables with Mean Ratings 
at or Above 2.7 by Total & Group 






Mean Rating by Stakeholder Groups 






TOTAL 

(1123) 


RER 

(61) 


SER 

(55) 


PSY 

(207) 


PRN 

(91) 


SD 

(37) 


C1D 

(41) 


RET 

(182) 


SED 

(449) 


Meta Cognitive Variables 




















self -regulatory, self-control strategies 


2.78 






X 


X 






X 


X 


Implementation. Classroom Instruction. & Climate Var 




















tables 




















Classroom Implementation Support 




































X 


X 


establish efficient classroom routines and communication 


170 


















Classroom Instruction 




















use of clear instruction 


173 




X 










X 


X 


teacher conveys enthusiasm 


2.76 






X 


X 






X 


X 


providing frequent feedback 


2.81 




X 


X 


X 


X 


X 


X 


X 


teaching for understanding 


111 






X 


X 




X 


X 


X 


good examples and analogies 


171 














X 


X 


Quantity of Instruction 




















time on task 


180 


X 


X 


X 


X 


X 


X 


X 


X 


time on reading skills 


170 




X 


X 


X 


X 




X 


X 


Classroom Management 




















teacher "withitness" (awareness) 


175 






X 


X 




X 


X 


X 


Student and Teacher Interactions (social) 




















teacher reacts appropriately 


175 






X 


X 




X 


X 


X 


Classroom Climate . - 




















task difficulty 


174 






X 


X 




X 


X 


X 


low apathy 


170 






X 


X 






X 


X 


organization 


180 






X 


X 






X 


X 



t 



i 

\ 





Table 3 

Patterns of Differences in Mean 
Ratings at or Below 1.9 Among Stakeholder Groups 



Mean Rating by Stakeholder Groups 



Variables with Mean Ratings TOTAL RER SER PSY PRN SD ClD RET SET 

at or Below 1.9 by Total & Group (1123) (61) (55) (207) (91) (37) (41) (182) (449) 



StAtr ar.d ntstrlrt Variables 

District Level Demographics 
school district size 

degree of school district centralization 
contractual limits on after school meetings 
contractual restrictions on aide activities 
efficiency of transportation system 

State Level Policy 

degree of state control of textbooks 

School L evel Variables 

School Level Demographics 

public vs. private school 
level of title VH (bilingual) funding 
. mix of student language backgrounds 

Student Variables 

Student Level Demomographics 

chronological age 

gender 

ethnicity 

Implementation. Cl assroom Instruction. A 
Climate Variables 

Quantity of Instruction 

time spent viewing educational TV 

Student and Teacher Interactions (academic) 

teacher asks questions low in difficulty 
teacher asks questions low in cognitive level 

Classroom Climate 

competition 



1.83 x 

1.83 x 

155 x 

1.83 x 

1.73 x 



1.74 x 



1.73 x 

1.89 x 

1.87 



1.86 x 

1.41 x 

1.62 x 



1.85 x 



1.77 x 

1.71 x 



1.82 x 



x 

Xxx 
X x X 

X 

X X 



X X 



X 



X X X X 



X 

X 



XXX 

XXX 

XXX 



X 



X X 



X 

X 

X 



X 

X 

X 



x X 

XXX 
X X X 



X 

X X 

X X 



X X 



X x x 



X X 

X X 



X X x x X 

X X X X x 



X X 



X X 



X 



Note: RER - regular ed. researcher 
SER - special ed. researcher 
PSY - psychologist 
PRN - principal 



SD - state director 
ClD • chapter 1 director 
RET - regular ed. teacher 
SET - special ed. teacher 
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Another caveat applies to the content analysis of research literature on 
group-level effects, notably the literature on effective schools. Some of the 
effective schools factors have been analyzed in relation to school averages on 
achievement tests. Such relationships might be found somewhat larger or 
smaller if calculated for individual children. It can be expected that expert 
reviewers on this subject (on which the syntheses depend) would take this 
uncertainty into consideration in interpreting their findings. It has rarely 
been demonstrated that techniques that work for the average student have 
deleterious consequences for other students’ learning. Nonetheless, it is worth 
keeping this limitation in mind in interpreting the findings and in tracing their 
implications. 

There are many other cautions that ordinarily apply to educational - 
research, such as the possibility that effective methods found a decade ago no 
longer apply today. These are obvious enough to leave to researchers and 
experienced educators as they think about how the findings apply in their own 
situations. While cognizant of the limitations of the CMOVS data base, several 
practical applications can be envisioned. 

The following section provides an example of how the CMOVS can be 
used as a guiding framework to improve schools’ programs. 

The Application of the CMOVS 

Implications of the use of the CMOVS’ to enhance communication among 
researchers and practitioners who make programmatic decisions and align 
studies on variables that are important to learning are manifold. One such 
application is the CMOVS’ provision of a "marker" system for describing 
program design and implementation features (effective practices) for schools 
aiming to improve student learning outcomes. 

As discussed in the previous section on the research base of the CMOVS, 
a total of 228 variables considered to be important to learning were culled 
from the research literature and based on consensus from the field. The 
variables were grouped under six major marker categories: a) state and 





district variables; b) out-of-school contextual variables; c) school-level 
variables; d) student variables; e) program design variables; and f) 
implementation, classroom instruction, and climate variables. Further 
delineations of the six categories of marker variables resulted in 30 sub- 
categories. The sub-categories under each of the major categories are listed 
in column 1 of Table 4. For example, two sub-categories of marker variables 
were identified under the category of state and district variables. They are: 
district-level demographics and state-level policy variables. Similarly, the 
category of out-of-school contextual variables consists of four marker 
variables: community, peer group, home environment and parental support, and 
student use of out-of-school time variables. 

The 30 sub-categories of the CMOVS can be used as a guideline for 
school improvement in a variety of ways. Table 5 provides an illustration of 
how the CMOVS can be useful to program developers, implementors, and policy 
makers for making informed decisions on selection of innovative 
practices/programs. The procedure involves systematically analyzing features 
of specific educational approaches/practices and assessing them in relation to 
needs of local schools. 

Column 1 of Table 4 shows a list of 30 categories of variables that are 
important to learning included in the CMOVS and the anticipated outcomes of 
the restructured program desired by a particular user. The second column 
shows the specific weightings of each variable category based on the consensus 
from the field (Wang, Walberg, Reynolds, and Rosenfield, 1989). The "X"s 
listed in each program column indicate that particular variables were 
considered in the design of a specific approach or practice being reviewed by 
a user. 

For example. Program B is a program designed using a teacher 
collaboration approach. Variables that were explicitly considered in the design 
of Program B under the category of State and District Variables include 
district level demographics variables and state level policy variables. Under 
the category of School Level Variables, teacher/administrator decision-making 
variables, school culture variables, and school-wide policy and organizational 



Table 4. Decision Making Framework: A Preliminary Analysis 
of tbe Programmatic Emphases of Selected Programs 





VARIABLE 


Program A 


Program B 


Program C 


Program D 


Program E 




WEIGHTING 


A Peer 


A Teacher 


A 


A Curriculum 


A Comprehensive, 




Based on 


Collaboration 


Collaboration 


Restructured 


Modification 


Integrated Education 


Marker Variable Categories 


Consensus 
from the Field* 


Approach 


Approach 


Classroom 

Approach 




and Related Service 
Delivery Approach 


I. VARIABLES CONSIDERED 














IMPORTANT TO LEARNING 














A, District Variables 














1. District Level Demographics 


(2) 




X 


X 




X 


Variables 














2. State Level Policy Variables 


(2) 




X 






X 


B. Out of School ConlcxauJ Variables 














1. Community Variables 


(2) 










X 
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Abstracted from Wang, Walberg, Reynolds and Rosenfield (1989), 

"Variables Important to Learning: A Consensus From the Field, " Temple University Center for Research in Human Development and Education. 
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Table 5. An Illustration of Using the Decision-Making Framework to Calculate Effectiveness Index 
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•Importance rating scale: 3 (high importance); 2 (moderate importance); 1 (low importance) 

••Effectiveness Rating: Variable Weighting x Importance Rating for variables emphasized in a given program 
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variables were considered important; but demographic variables, accessibility 
variables, and parental involvement policy variables were not emphasized in the 
design of Program B. 

Based on the variables considered important (shown in Table 4), several 
simple quantitative indexes can be generated as a basis for making program 
design decisions. For example, these indexes can be used to develop an 
information base for identifying program development needs and/or selecting a 
particular approach or practice for adoption or adaptation in order to meet the 
improvement needs of a particular school. Examples of the variety of indices 
that can be generated for consideration in making programming decisions are 
presented below. 

Program Effectiveness Index 

Using the variable weightings based on the consensus from the field 
(shown in column 2 of Table 5), plus the information on design features 
emphasized in the various programs as indicated by "X"s in Table 5, potential 
users can develop an effectiveness index that reflects site-specific needs as 
they make selection judgments on given approaches or practices. 

Calculating the Program Effectiveness Index . The first step in developing 
a Program Effectiveness Index is to calculate the importance rating by the 
user (potential adopter of the program/approach). This is done by asking the 
user to rate the importance of the variable categories listed in Column 1 of 
Table 5, using a three-point scale. A rating of "3" indicates that a particular 
variable category is considered of high importance in terms of the user’s site- 
specific needs; a rating of "2" indicates that a particular variable category is 
of moderate importance; and a rating of "1" indicates that a particular variable 
category is of low importance. Users’ ratings may be based on a variety of 
user-specific information (e.g., their own experiences, current programs 
implemented in their respective schools, knowledge of a particular set of 
research findings, philosophical alliances or differences on a specific 
instructional approach, and the importance of the variables from their own 
site-specific perspectives). The quantitative index derived from the potential 
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users’ importance ratings will enable them to make decisions on the extent to 
which the various educational approaches and program specific practices of the 
various extant programs being considered meet the program improvement and 
implementation support needs of their respective schools/school districts. 

Table 5 provides an example of how a hypothetical user can apply the 
CMOVS for calculating a Program /Effectiveness Index. The second column of 
Table 5 shows the Variable Weighting Scores (based on the three-point scale) 
of each of the variable categories included in the CMOVS. These scores are 
the result of consensus ratings from the field (Wang, Reynolds, Walberg & 
Rosenfield, 1989). The hypothetical user’s importance rating of each of the 
variables included in the CMOVS are listed in Column 3 of Table 5. The 
number listed in the last row of column 3 is "79," the total possible 
Importance Score (the users’ judgments on the importance or relevance of the 
CMOVS variables to the educational goals and/or program improvement needs 
of their specific schools/school districts). 

As an illustration, columns S and 7 show the program effectiveness 
ratings for Program A and Program B respectively. For example, the particular 
hypothetical user was interested in adopting either Program A, which uses the 
peer collaboration approach, or Program B, which uses a teacher collaboration 
approach. The hypothetical user calculated Program Effectiveness Indexes for 
Program A and Program B based on the ratings of variable categories 
considered important for meeting her/his program improvement and/or 
implementation support needs. As shown in the last row of column S, the 
program’s overall Effectiveness Index for meeting the site-specific improvement 
needs for Program A is "91." This score is the sum of the user’s Importance 
Ratings for each of the variable categories emphasized in the design of 
Program A (indicated by an "X") multiplied by the corresponding Variable 
Weighting Scores based on the consensus from the field (column 2). For 
example, the "Program Effectiveness Index" for variable category B.2 (Peer 
Group Variables) of Program A equals "4." The Program Effectiveness Index 
for variable category B.2 is derived by multiplying a Variable Weighting of "2" 
x an Importance Rating of "2" x "1" (the fact that this variable is emphasized 
in the design of Program A as indicated by an "X"). 



O 

ERiC 



22 



13 



Thus, based on the overall Effectiveness Scores, as shown in Table 3, 
without considering other factors. Program A (with a score of "91") seems to 
match the particular hypothetical user’s program improvement needs better 
than Program B (with a score of "57"). 

Desirability of Implementation Index 

Another way of using the CMOVS for making program design decisions is 
to calculate a Desirability of Implementation Index for the approaches/practices 
being considered. Some variables may be relatively easy to incorporate in the 
on-going program at a given school for various reasons, (e.g., because the 
approach or practice being considered has already been incorporated into their 
programs, the staff can be quickly trained to implement the variables on a 
systematic basis). Other variables, however, may require extensive training, 
special materials, and modifications of the overall school program. Thus, the 
implementation of a given program or a component of a program may require 
special techniques or implementation support not as easily or feasibly 
integrated into the ongoing programs. Therefore, implementation may not be 
as "desirable" for a given user’s specific situation. The Desirability of 
Implementation Index is calculated according to the user's judgement on the 
feasibility of implementing the program in his or her school. This is 
determined by using the Feasibility of Implementation Rating and the Variable 
Weighting based on the consensus from the field. 

Calculating. -the .Feasibility o f Implementation Rating The purpose of 
calculating the Feasibility of Implementation Index is to quantify the extent to 
which implementation of a selected approach/practice is feasible. The 
weighting method used in the development of the Program Effectiveness Index 
discussed above can also be used in calculating the feasibility of implementing 
an approach/practice and/or a particular component of a program. Based on a 
three-point scale, users first determine a Feasibility of Implementation (instead 
of importance) Rating by assigning a value to each variable category included 
in the CMOVS. A rating of "3" is given to a variable category that could be 
easily implemented based on the user’s judgement; a rating of "2" is given to a 
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variable category that could be moderately implemented; and a rating of ”1” is 
given to a variable category that could prove difficult to implement. The 
Feasibility of Implementation Rating Scores for each of the variable categories 
are listed in column 3 of Table 6. 

Calculating the Desirability of Implementation Index. Once the Feasibility 
of Implementation Rating Scores have been calculated, they may be used to 
calculate the Desirability of Implementation Index for a given program. To 
obtain a Desirability of Implementation Index (see columns 5 and 7 of Table 6), 
the Feasibility of Implementation Rating for each variable considered in the 
design of a particular program (indicated by an ”X”) is multiplied by its 
corresponding Variable Weighting Score based on consensus from the field 
(column 2). By adding the Desirability of Implementation Ratings for each of 
the variable categories considered, the user is able to derive an overall 
Desirability of Implementation Score for each given program (see the last row 
of columns S and 7 in Table 6). This index essentially reflects both the 
consensus from the experts on variables that are important to learning and the 
particular program’s "desirability'' for implementation based on the users’ best 
judgments of feasibility in the context of site-specific circumstances. 
According to Table 6, Program A, which has a Desirability Score of 101, is 
probably a more feasible program for the user’s purposes as compared to 
Program B, which has a Desirability Score of 58. 

Conclusion 

There are multiple ways to handle the mathematics for calculation of the 
various indexes. Users may design other indexes to meet their own needs. 
The necessary evaluation can be done by hand or by using a computer program 
with a built-in weighting index. Depending on the' intended purpose, the user 
determines which index to use and how to apply the information derived from 
the various indexes. Appendix B includes a diagram of the schematic process 
for the computer analysis program, a sample of the computerized interactive 
decision-making program, and the results of the computer analysis of the 
user’s ratings of a given approach/practice being considered. The printout 
shows a suggested list of candidate approaches/practices that include features 
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Table 6. An Illustration of Using the Decision-Making Framework to Calculate Desirability of Implementation Index 
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•Implementation rating scale: 3 (high importance); 2 (moderate importance); 1 (low importance) 
••Stakeholder Rating: Variable Weighting x Importance Rating for variables emphasized in a given program 
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that arc most responsive to variables considered important to learning. 

The use of the CMOVS as a decision-making framework for developing a 
site-specific program effectiveness and feasibility data base has several virtues. 
Using a quantification methodology to derive a data base for decision-making 
will allow the user to have multiple information resources that are systematic 
and specific to their information needs. Users may adopt the average ratings 
as calculated, or develop their own weighting schemes. They can combine this 
information with their best judgment of their own situations and the 
characteristics of the students they serve. 

Although the foregoing discussion emphasizes the use of the framework 
by potential consumers of educational programs, it may also prove useful to 
curriculum designers and developers of innovative programs. The list of 
variables included in the CMOVS can serve as a checklist to determine which 
variables are critical to consider in program development and evaluation 
efforts. The checklist ensures that the program design incorporates features 
that research suggests are important to enhance learning efficiency and 
productivity. Thus, from the outset, consideration can be given to the variety 
of ways in which approaches or practices can be implemented. 

If all programmatic factors were equal, it could be anticipated that the 
fully implemented programs which include more significant variables (features) 
would improve learning the most. In actual practice, however, all the factors 
involved are unlikely to be equal. Programs with extensive features are likely 
to be more costly to implement and manage. Therefore, both program 
developers and users need to carefully analyze the site-specific constraints and 
needs and weigh the trade-offs between cost and effectiveness in identifying 
priorities and in making programmatic decisions. 
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Variables Important To Learning: 
A Consensus From the Field 
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A Summary of Findings from 
A Survey of Consensus from the Field 



Variables 



Consensus Rating 



Category I. State and D istrict Variables: 



These are variables associated with state and district level school governance and 
auiiiinisUauCn. They include state curriculum and textbook policies, testing and 
graduation requirements, and teacher licensure; as well as specific provisions in 
teacher contracts, and some district-level administrative and fiscal variables. 
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I-B. 
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School district size 
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Degree of school district centralization 

Presence of contractual limits on after-school meetings 

Limits on class size 

Presence of contractual restrictions on activities performed by aides 
Degree of central office assistance and support for programs 
Degree of board of education support for instructional programs 
Per pupil expenditure 
Efficiency of transportation system 



State Level Policy Variables 
1 . 

2- Teacher licensure requirements 

3- Degree of state control over textbooks 

4 - Degree of state control over curriculum 
Academic course and unit requirements 

6- Minimum competency test requirements 

Adherence to least restrictive environment/mainstreaming 















Note: ** = highly important (mean rating of 2.6 and above, based on a 3-point scale) 

* = moderately important (mean rating of 2.0 • 2.5, based on a 3-point scale) 
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Variables 



Consensus Raring 



Category II. Out of School Contextual Variables: 

These are variables associated with the home and community contexts within 
which schools function. They include community demographics, peer culture, 
parental support and involvement, and amount of time students spend out-of- 
school on such activities as television viewing, leisure reading, and homework. 

II-A. Community Variables 

1. Socioeconomic level of community ** 

2. Ethnic mix of community * 

3. Quality of social services for students * 

EI-B. Peer Group Variables 

1 ■ Level of peers' academic aspirations *• 

2. Level of peers' occupational aspirations •• 

3. Presence of well defined clique structure * 

4. Degree of peers' substance abuse •• 

5. Degree of peers' criminal activity •• 

II-C. Home Environment and Parental Support Variables 

1 . Educational environment (e.g., number of books and magazines at home) • • 

2. Parental involvement in assuring completion of homework •• 

3. Parental ;r"Olvement in assuring regular school attendance ** 

4. Parental monitoring of student television viewing •• 

5. Parental participation in school conferences and related activities • 

6. Parental application of appropriate, consistent discipline ** 

7. Parental expression of attention to children •• 

8. Parental interest in student's school work •* 

9. Parental expectation for academic success •• 

D-D. Student Use of Out of School Time Variables 

1. Student participation in clubs and extracurricular school activities 

2. Amount of time spent on homework • 

3. Amount of time spent on leisure reading • 

4. Amount of time spent viewing educational television 

5. Amount of time spent viewing noneducarional television • 



Variables 



Consensus Rating 



Category m . School Level Variables: 

These are variables associated with school-level demographics, culture, climate, 
policies, and practices. They include demographics of the student body, whether 
the school is public or private, levels of funding for specific categorical pro- 
grams, school-level decision making variables, and specific school-level policies 
and practices, including policies on parental involvement in the school. 

HI- A. Demographic and Marker Variables 

1. Public versus private school 

2. Size of school 

3. Level of Chapter I (compensatory education) funding * 

4. Level of Title VD (bilingual) funding 

5. Level of PL 94- 142 (handicapped) funding * 

6. Mix of socioeconomic levels in the school * 

7. Mix of cultural/ethnic groups in the school 

8. Mix of student language backgrounds in the school 

m-B. Teacher /Administrator Decision Making Variables 

1 . Teacher and administrator consensus on school values, norms, and ** 

roles 

2. Principal actively concerned with instructional program •* 

3. Teacher involvement in curricular decision making •• 

4. Teacher involvement in instructional decision making ** 

5. Teacher involvement in resource allocation decisions * 

6. Teacher involvement in finding ways to increase academic ** 

performance 

m-C. School Culture Variables (Ethos Conducive to Teaching and 
Learning) 

1 . Use of cooperative, not exclusively competitive, goal structures * 

2. School-wide emphasis on and recognition of academic achievement •• 

3. Low staff absenteeism * 

4. Low staff turnover 

5. Low staff alienation ** 

6. Active collaboration between regular classroom teachers and special •• 

education teachers 

7. Safe, orderly school climate «« 

8. Degree of school personnel professional collaboration •• 





Variables 


Consensus Rating 


Catepnrv III. Crh™i l evel Variables: (continued) 




ra-D. 


School-Wide Policy and Organizational Variables 






1 . Presence of "effective schools program" 


• 




2. Explicit school grading and academic progress policies 


• 




3 . Explicit school* wide discipline policy 


• 




4 . Explicit school-wide attendance policy 


• 




5. Coordination of pullout programs for handicapped students with 

regular instructional programs 

6. Use of multi-age grouping 


• 




7. Use of instructional teaming 


• 




8. Use of cross-age tutoring 


• 




9. Use of peer tutoring 


• 




10. Use of academic tracking for specific school subject areas 

1 1 . Minimization of external classroom disruptions (e.g., broadcast 

announcements) 


• 




12. Adherence to least restrictive environment/mainstreaming 


• 




13. Minimum use of suspension and expulsion as discipline tools 


• 


ra-E. 


Accessibility Variables 






1. Accessibility of educational program (overcoming architectural, 
communication, and environmental barriers 


* 


ra-F. 


Parental Involvement Policy Variables 






1 . Parental involvement in improvement and operation of instructional 
programs 


* 




2. School-sponsored parenting skills workshops (e.g., behavior 
modification, parent effectiveness training) 


• 
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Variables 



Consensus Rating 



Category I V. Student Variables: 

These are variables associated with individual students themselves, including 
demographics, academic history, and a variety of social, behavioral, motivational, 
cognitive, and affective characteristics. 

IV- A. Demographic and Marker Variables 

1. Chronological age 

2. Socioeconomic status * 

3. Gender 

4. Ethnicity 

5. First or native language * 

6. Physical and health status * 

7 . Special education classifications (e.g., EMR, LD) * 

IV-B. History of Educational Placements 

1 . Prior grade retentions * 

2. Prior special placements • 

3. Current placement in regular class versus self-contained special * 

education class 

IV-C. Social and Behavoriai Variables 

1. Positive, nondisruptive behavior •• 

2. Appropriate activity level •• 

3. Cooperativeness with teacher •• 

4. Cooperativeness with peers •• 

5. Ability to make friends with peers • 

IV-D. Motivational and Affective Variables 

1. Attitude toward school •• 

2. Attitude toward teachers •• 

3. Attitude toward subject matter instructed •• 

4. Motivation for continual learning ** 

5. Independence as a learner •• 

6. Perseverance on learning tasks •• 

7. Self-confidence 

8. Academic self-competence concept in subject area instructed • • 

9. Attributions for success and failure in subject area instructed « • 





Variables 


Consensus Rating 


Category IV. Student Variables: (continued) 




IV-E. 


Cognitive Variables 






1 . Piagetian stage of cognitive development 


* 




2. Level of reasoning (fluid ability) 


** 




3. Level of spatial ability 


* 




4. Memory 


** 




5. Level of general academic (crystallized) knowledge 


** 




6. Level of specific academic knowledge in subject area instructed 


* 




7. Level of reading comprehension ability 


** 




8. Level of writing ability 


* 




9. Level of computational ability 


* 




10. Level of oral fluency 


* 




1 1 . Level of listening skills 


• * 




12. Learning styles (e.g., field independent, visual/auditory learners, 
high cognitive complexity) 


* 


IV-F. 


Metacognitive Variables 






1. Self-regulatory, self-control strategies (e.g., control of attention) 


** 




2. Comprehension monitoring (planning: monitoring effectiveness of 
attempted actions; monitoring outcomes of actions; testing, 
revising, and evaluating learning strategies) 


mm 




3. Positive strategies for coping with failure 


mm 




4. Positive strategies to facilitate generalization of concepts 


mm 


1V-G. 


Psychomotor Variables 






1 . Psychomotor skills specific to area instructed 


m 
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Variables 



Consensus Rating 



Category V. Program Design Variables: 

These are variables associated with instruction as designed, and with the physical 
arrangements for its delivery. They include the instructional strategies specified 
by the curriculum, and characteristics of instructional materials. 

V-A. Demographic and Marker Variables 

1 . Size of instructional group (whole class, small group, one-on-one * * 

instruction) 

2. Proportion of students with special needs served in regular classes 

3 . Number of classroom aides required 

4. Resources needed 

V-B. Curriculum and Instructional Variables 

1 . Clearly presented academic, social, and altitudinal program goals/ * 

outcomes 

2. Use of explicit goal/objective setting for instruction of individual 

student (e.g.. Individualized Educational Plans (IEPs) 

3. Use of mastery learning techniques, including use of instructional * 

cues, engagement, and corrective feedback 

4. Use of cooperative learning strategies 

5. Use of personalized instructionial program 

6. Use of prescriptive instruction combined with aspects of infonnal or 

open education 

7. Use of diagnostic -prescriptive methods 

8. Use of computer-assisted instruction 

9. Use of crisis management techniques to control classroom 

disrupt] veness 

10. Use of program strategies for favorable affective climate 

11. Alignment among goals, contents, instruction, assignments and * 

evaluation 

12. Curriculum units integrated around key discipline-based concepts 

13. Use of multidisciplinary approaches to instructional planning 

(including diagnosis in educational planning) 

14. Presence of information in the curriculum on individual differences 

and commonalities (including handicapping conditions) 

15. Presence of culturally diverse materials in the curriculum 
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Variables 



Consensus Raring 



Category V. Program Design Variables : (continued) 



V-C. Curriculum Design Variables 

1 . Materials employ alternative modes of representation * 

2. Material is presented in a cognitively efficient manner ** 

3. Materials employ explicit and specific objectives * * 

4. Materials employ advance organizers • 

5. Materials employ learning hierarchies * 

6. Materials are tied to assessment and diagnostic tests * 

7. Availability of materials and activities prepared specifically for use * 

with whole classroom, small groups, or one-on-one instruction 

8. Degree of structure in curriculum accommodates needs of different ** 

learners 

9. Student interests guide selection of a significant portion of content * 

10. Availability of materials and activities for students with different ** 

abilities 

1 1 . Availability of materials and activities for students with different * * 

learning styles 

12. Developmental issues considered * 

13. Student experiences considered • 



Variables 



Consensus Rating 



Category VI Implementation. Classroo m Instruction, and Climate Variables; 

These are variables associated with the implementation of the curriculum and the 
instructional program. They include classroom routines and practices, character- 
istics of instruction as delivered, classroom management, monitoring of student 
progress, and quality and quantity of instruction provided, as well as student- 
teacher interactions and classroom climate. 

VI -A Classroom Implementation Support Variables 

1 . Creation and maintenance of necessary instructional materials * 

2. Adequacy in the configuration of classroom space * 

3. Availability of classroom aides * 

4. Use of written records to monitor student progress 

5 . Establishing efficient classroom routines and communicating rules 

and procedures 

6. Developing student self-responsibility for independent study and ** 

planning of one’s own learning activities 
M-B Classroom Instructional Variables 

1 . Prescribing individualized instruction based on perceived match of * 

type of learning tasks to student characteristics (e.g., ability, learning 

style) 

2. Use of procedures requiring rehearsal and elaboration of new * 

concepts 

3. Use of clear and organized direct instruction ** 

4. Systematic sequencing of instructional events and activities ** 

5. Explicit reliance on individualized educational plans (IEPs) in * 

planning day-to-day instruction for individual students 

6. Use of instruction to surface and confront student misconceptions * 

7. Use of advance organizers, overviews, and reviews of obejctives to * 

structure information 

8. Clear signaling of transitions as the lesson progresses * 

9. Significant redundancy in presentation of content * 

10. Teacher conveys enthusiasm about the content ** 

11. Directing students' attention to the content ** 

12. Using reinforcement contingencies ** 

13. Setting and maintaining clear expectations of content mastery ** 

14. Providing frequent feedback to students about their performance ** 

15. Explicitly promoting effective metacognitive learning strategies * 

16. Promoting learning through student collaboration (e.g., peer tutoring, * 

group work) 

17. Corrective feedback in event of student error ** 

18. Flexible grouping that enables students to work to improve and ** 

change status/groups 

19. Teaching for meaningful understanding ** 

20. Degree to which student inquiry is fostered ** 

21 . Scaffolding and gradual transfer of responsibility from teacher to * 

student 

22. Degree to which assessment is linked with instruction * 

23. Skills taught within the context of meaningful application ** 

24. Good examples and analogies to concretize the abstract and ** 

familiarize the storage 

25. Consideration of the teacher’s use of language in the instructional ** 

process 

26. Explicitly promoting student self-monitoring of comprehension ** 



Variables 



Consensus Rating 



Category VI. Implementati on. Classroom Instruction, and Climate Variables: 

(continued) 

VI-C. Quantity of Instruction Variables 

1. Length of school year 

2. Length of school day • 

3. Time on task (amount of time students are actively engaged in •• 

learning) 

4. Time spent in direct instruction on basic skills in reading ** 

5. Time spent in direct instruction on basic skills in mathematics ** 

6. Time allocated to basic skills instruction by regular classroom ** 

teacher 

7. Time allocated to basic skills instruction by special education teacher •• 

8. Difference between academic learning time and allocated learning • 

time 

9. Time spent out of school on homework • 

10. Time spent out of school viewing educational television • 

1 1. Time spent out of school in informal learning experiences (e.g., • 

museum trips, scouts) 

12. Nature of regular classroom content missed by students during * 

participation in pullout programs 

Vl-D. Classroom Assessment Variables 

1. Use of assessments to create detailed learner profiles rather than * 

simple classifications or unlaborated total scores 

2. Use of assessment as a frequent, integral component of instruction * 

3. Accurate, frequent measurement of basic skills in reading * 

4. Accurate, frequent measurement of basic skills in mathematics * 

VI-E. Classroom Management Variables 

1. Minima] disruptiveness in classroom (e,g„ no excessive noise, no ** 

students out of place during instructional activities, no destructive 
activities) 

2. Group alerting (teaching uses questioning/recitation strategies that ** 

maintain active participation by all students) 

3. Learner accountability (teacher maintains student awareness of ** 

learning goals and expectations) 

4. Transitions (teacher avoids disruptions of learning activities, brings ** 

activities to a clear and natural close, and smoothly initiates new 
activity) 

5. Teacher "withitness" (teacher is continually aware of events and ** 

activities and minimizes disruptiveness by timely and nonconfron- 
tationaJ actions) 
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Variables 



Consensus Raxing 



Category VI. implementati on. Classroom Instruction, and Climate Variables: 

(continued) 

VI*F. Student and Teacher Interactions: Social Variables 

1. Student initiates positive verbal interactions with other students and * 

with teacher 

2. Student responds positively to questions bom other students and •• 

from teacher 

3. Teacher reacts appropriately to correct and incorrect answers ** 

4. Teacher reinforces positive social interactions with students rejected ** 

by peers 

5. Teacher provides explicit coaching on appropriate social behaviors ** 

6. Teacher provides explicit coaching to reduce aggression * • 

Vl-G. Student and Teacher Interactions: Academic Variables 

1 . Teacher asks academic questions frequently ** 

2. Teacher asks questions predominantly low in difficulty * 

3. Teacher asks questions that are predominantly low in cognitive level • 

4. Teacher maintains high post-question wait time • 

5. Frequent calls for extended, substantive oral and written response • 

(not one-word answers) 

VI-H. Classroom Climate Variables 

1. Cohesiveness (members of class are friends sharing common * 

interests and values and emphasizing cooperative goals) 

2. Low friction (students and teacher interact in a considerate and ** 

cooperative way, with minimal abrasiveness) 

3. Low cliqueness (students work with many different classmates, and * 

not just with a few close friends) 

4. Satisfaction (students are satisfied with class activities) ** 

5. Speed (the pacing of instruction is appropriate for the majority of the •• 

students) 

6. Task difficulty (students are continually and appropriately ** 

challenged) 

7. Low apathy (class members are concerned and interested in what ** 

goes on in the class) 

8. Low favoritism (all students are treated equally well in the class, and «« 

given equal opportunities to participate) 

9. Formality (students are asked to follow explicitly stated rules • 

concerning classroom conduct and activities) 

10. Goal direction (objectives of learning activities are specific and ** 

explicit) 

11. Democracy (all students are explicitly involved in making some * 

types of classroom decisions) 

12. Organization (class is well organized and well planned) ** 

13. Diversity (the class divides its efforts among several different • 

purposes) 

14. Environment (needed or desired books and equipment are readily •• 

available to students in the classroom) 

15. Competition (students compete to see who can do the best work) * 
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CMOVS Computer Analysis: 

User Ratings of Selected, Innovative Educational Practices 
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Sample Screens From the Computerized Interactive Decision-Making Program 



Screen #1 



In this computer application, you will be asked to rate the importance and feasibility of 30 variables according to your 
site-specific circumstances. This will enable you to calculate specific indexes to help you in your programmatic deci- 
sion-making. 

HIT RETURN TO CONTINUE 



Screen #2 



To begin, rate the importance of the following State and District Variables by typing a 0, 1, 2, or 3. 
HIT RETURN TO CONTINUE 



Screen #3 



1 . District Level and Demographic Variables 






0 


i 


2 


3 


* 1 

(not sure) 

HIT RETURN TO CONTINUE 


1 

( not important) 


1 

(somewhat important) 


1 

(very important) 



Screen #4 



2. Stale Level Policy Variables 
0 


i 


2 


3 


1 

(not sure) 

HIT RETURN TO CONTINUE 


i 

( not important) 


1 

(somewhat important) 


1 

(very important) 



CMOVS Computer Analysis: 

User's Effectiveness, Feasibility, and Desirability Ratings 
of Approaches/Practices 



EFFECTIVENESS RATING 



The following innovative educational appro aches/pracrices are recommended in order of 
their effectiveness from the highest to the lowest according to the user's ratings: 



Using a comprehensive & integrated approach to service delivery 
score ----- - — - -- -- -- -- - — ............ 




Using a restructured classroom approach 

score — 


- - - 120 


Using a peer collaboration approach 

score — - - - - — - - ------------- 




Using a teacher collaboration approach 

score 


--•62 


Using a curriculum modification approach 

score 




FEASIBILITY RATING 





The following innovative educational approaches/practices are recommended in Older of 
their feasibility from the highest to the lowest according to the user's ratings: 



Using a comprehensive & integrated approach to service delivery 
score ~ " 


■-•146 


Using a restructured classroom approach 

score “ ~ ~ 


' " “120 


Using a peer collaboration approach 
score -------------------- 


• 

00 

o 


Using a teacher collaboration approach 

score " 




Using a curriculum modification approach 

score 




DESIRABILITY RATING 





The following innovative educational approaches/practices are recommended in order of their 
desirability from the highest to the lowest according to the user's ratings: 



Using a comprehensive & integrated approach to service delivery 
scare 


676 


Using a restructured classroom approach 
scare 


568 


Using a peer collaboration approach 
scare 


"’428 


Using a teacher collaboration approach 
scare 


* 292 


Using a curriculum modification approach 
scare 


276 
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